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Abstract 

In order to attain a reliable laboratory work assessment, we argue taking the Learning Environment as a core 
concept and a research paradigm that considers the factors affecting the laboratory as a particularly complex 
educational context. 

With regard to Laboratory Learning Environments (LLEs), a well known approach is the SLEI (Science Laboratory 
Environment Inventory). The aim of this research is to design and apply an alternative and qualitative 
assessment tool to characterize Laboratory Learning Environments in an introductory course of organic 
chemistry. An alternative and qualitative assessment tool would be useful for providing feed-back for 
experimental learning improvement; serving as a complementary triangulation tool in educational research on 
LLEs; and generating meaningful categories in order to design quantitative research instruments. 

Toward this end, spontaneous questions by students have been chosen as a reliable source of information. To 
process these questions, a methodology based on the Grounded Theory has been developed to provide a 
framework for characterizing LLEs. This methodology has been applied in two case studies. The conclusions 
lead us to argue for using more holistic assessment tools in both everyday practice and research. Likewise, a 
greater attention should be paid to metacognition to achieve suitable self-perception concerning students' 
previous knowledge and manipulative skills. 

Keywords - Learning environments, metacognition, autonomy, science laboratory, student questions. 


1 INTRODUCTION 

1.1 Laboratory work from a comprehensive viewpoint of educational assessment 

A comprehensive viewpoint of educational assessment emphasizes its role as a tool for learning improvement. 
This is particularly relevant with regard to laboratory activities since these involve many different specific 
competences related to the following phases (Hofstein, 2004): (a) planning and design, (b) performance; (c) 
analysis and interpretation and (d) application. The execution of each of these activities is highly influenced by 
countless contextual issues. Likewise, several generic competences have significant weight in laboratory 
activities, particularly those concerning communication skills, cooperative work, and decision making. In 
addition, the influence of infrastructural issues concerning organization and management of educational 
resources are much greater than those involved in ordinary classroom activities. 
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This complexity is a challenge that requires a new conceptual approach. Top priority should be placed on 
developing an accurate characterization of laboratory Learning Environment. In fact, we argue for taking this 
concept as a key issue and unit of analysis both in ordinary laboratory work assessment and educational 
research. 

1.2 Laboratory Learning Environment as a key concept to analyze the experimental work 

As discussed by Wilson (1995), the Learning Environment concept is a coherent, valid and reliable conceptual 
approach for: a) Carrying out an authentic and comprehensive assessment in ordinary learning (Bravo & 
Fernandez del Valle, 2000), b) Designing research instruments to test innovative educational proposals. 

Designing instruments to test educational models is particularly relevant in our educational context, as it is still 
strongly influenced by a conception of laboratory work taks as decontextualized hands-on activities, or simple 
testing of taught theoretical knowledge. Indeed, they are usually developed as a parallel learning process, more 
or less coordinated with the theoretical syllabus, and seldom explicitly integrated in a coherent learning 
sequence. This trend coexists with many innovative experiences whose goal is to correct these methodological 
deficiencies Thus, monitoring these changes in learning environments requires specific and suitable assessment 
tools. 

The characterization of laboratory learning environments (LLEs, from now on) cannot be restricted to basic 
issues concerning infrastructure or educational resources. The same is true with organizational models related 
to interactions between students and teachers, and among students themselves. From a constructivist point of 
view, issues related to human and affective development, or to relationships in social and professional contexts 
should be taken into account. Powerful or rich learning environments have been described by several authors 
as Dochy, Segers, Van den Bossche and Struyven (2005) and Gijbels, Van den Watering, Dochy and Van den 
Bossche (2006). Moreover, attention paid to autonomy enhancement is especially noteworthy. In that respect, 
personal responsibility, initiative development, and decision making ability etc., are central issues in improving 
learning. 

Many tools have been proposed to characterize the Learning Environments, such as CLES (Constructivist 
Learning Environment Survey) (Fraser, 2002). Some of the most representative tools-- LEI (Learning 
Environment Inventory) or CES (Classroom Environment Scale)-- are based on quantitative questionnaires. 
Nevertheless, other tools combine qualitative and quantitative methods (Fraser, 2002). The nature and goals 
of LLEs lead us to consider the Mixed Methods Research paradigm, widely described by Onwuegbuzie and 
Leech (2005). The main feature of this emerging paradigm is a pragmatic approach based on the integration 
between qualitative and quantitative methods. 

Regarding LLEs, the SLEI (Science Laboratory Environment Inventory) is an extensively used tool, which has 
been applied in many research studies using large samples of students Hofstein, Nahum and Shore (2001). This 
assessment tool consists of objective questionnaires whose items include the following scales: (a) Students' 
cohesiveness; (b) Activity open-endedness; (c) Integration; (d) Clarity rules; (e) Infrastructure. 

1.3 The characterization of Laboratory Learning Environments from a Mixed Research 
Paradigm: Student generated questions as a useful data source 

From the Mixed Research Paradigm, we argue for using triangulation as a strategy based on combining 
different methodologies (Fraser, Gidding & McRobbie, 1992). Indeed, there is experimental evidences of the 
unreliability of LLEs assessment tools based only on students' survey responses. According to Richardson 
(2004), questionnaire- based research should consider possible response bias in such instruments. Results may 
be influenced by affective issues as well as by students' general understanding and vision of learning activities. 
Indeed, the influence of cultural issues and personal traits in surveys results has been studied extensively in 
order to identify "response styles" (O'Neill, 1967). 

A promising, alternative approach to achieving triangulation may be the analysis of spontaneous, students- 
generated questions (SGQs, from now on) during laboratory sessions (Harper, Etkina & Lin, 2003; Teixeira-Dias, 
Pedrosa de Jesus, Neri de Souza & Watts, 2005). According to Teixeira-Dias et al. (2005), SGQs can be 
considered an interactivity indicator, showing the degree in which the students organize their activities in a 
cooperative learning environment (Chin & Osborne, 2008). Furthermore, given the amount and quality of 
interactions are considered the most relevant features of an inquiry learning environment, SGQs analysis can 
be a useful way to characterize it. As proposed by Pedrosa de Jesus, Neri de Souza, Teixeira-Dias and Watts 
2004), two general types of questions can be distinguished: questions that allow students to manage their 
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cognitive process themselves-showing ability and autonomy development—and operative questions involving 
concrete tasks organization and manipulative procedures. From the standpoint of laboratory learning, the 
following reflections justify the usefulness of the SGQs analysis: 

• SGQs are a reliable expression of greatest or lowest acquisition of previous necessary knowledge, 
abilities and skills. In addition, the degree to which information requested in SGQs pertains to laboratory 
work-- prelab content, laboratory manual, advice regarding equipment use, etc. (information which must be 
known by students without receiving any help)-- may be a good indicator of student engagement. At the 
same time, SGQs analysis may be useful for evaluating educational resources (laboratory manuals, 
videotape recordings, etc.). Likewise, some aspects concerning to laboratory organization and management 
such as rule clarity, material and equipment availability, wastes storage, etc., can be analysed. 

• SGQs can help us analyse curricular integration of experimental activities, such as the suitability of 
prelabs. 

• SGQs can be useful indicators to identify the learning problems related to lab work methodology. SGQs 
may reveal the degree to which experimental tasks are developed as an inquiry and reflective process, or a 
"cookbook" approach. 

• Lastly, SGQs reflect students' ability to distinguish what is important from what is accessory. Thus, 
irrelevant aspects of concepts or experimental procedures are often overrated, hiding the most meaningful 
work issues. This conceptual and operational overload can be related to what has been referred to in the 
literature as "noise" (Byrne, 1990). 

1.4 Research Aims 

This research aims to develop and apply a methodology based on SGQs analysis in order to make an overall 
assessment of laboratory learning environment as a tool for decision-making in the design and management of 
laboratory work activities. The final outcome of this analysis is to create an encoding framework to classify and 
interpret SGQs in order to be able to quantify in a simple descriptive way the most meaningful educational 
categories. This tool should be useful in the following ways: a) as a triangulation research strategy when used 
together with quantitative questionnaires; b) for carrying out comparisons among different educational or 
alternative learning contexts; and c) for monitoring implementation of educational innovations activities, etc. 
from a holistic point of view. 


2 DESIGN AND METHODOLOGY 
2.1 Context and design 

This experiment was carried out in the first course at the Technical Superior School of Rural Environments and 
Enology, at the Polytechnic University of Valencia (Spain). The analysed laboratory sessions are part of a four- 
month course in introductory organic chemistry. The laboratory activities take place in 2-hour long sessions. 
Attending student numbers have fluctuated between 8 and 26, though in most cases they have hovered 
around 15. Prelab activities consist of an audio visual tutorial together with an online test (Llorens-Molina, 
2008). 

A total amount of 377 SGQs from 25 laboratory sessions were collected over five consecutive years. The 
working method begins with writing down immediately, in situ, the SGQs addressed to teacher during 
laboratory sessions. Sometimes very short explanations and marginal notes are added to the SGQ text in order 
to facilitate the encoding process. 


2.2 Methodology 


To make sense of SGQs, a qualitative research design based in the Grounded Theory (Glaser & Strauss, 2006) 
has been developed by means of an encoding process, developed by Charmaz (2006). From this theoretical 
point of view, the key process for turning data into theory is the stepwise refinement of the encoding 
framework. It concerns both code texts (they become progressively more inclusive and educational 
meaningful) and their relationships (progressively more diversified and complex). The encoding process starts 
with theoretical sampling and data collection, followed by initial encoding, a focused encoding, and ends with 
the making and hierarchical structuring of theoretical categories. The encoding process was carried out by 
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means of software Atlas.ti (5.2) (Qualitative data analysis, 2002-2012). Below, this process is described in 
detail. 

The starting material (primary documents) is the teacher report containing SGQs and their marginal notes. Each 
question (or a part to which can be attributed an independent meaning) is considered a quotation. From these 
quotations, an open encoding process leads to a set of initial codes. These may be keywords or language 
structures from quotations ("in vivo" encoding). For example: 

• "What do we must do now?" 

• "How do I switches on the heater?" 

• "Where is the paper filter?" 

• "The ice is added for it to precipitate, isn't it?" 

• "Where are the oil wastes thrown out?" 

Likewise, other initial codes may come from primary categories with educational meaning. For example: 

• The laboratory task goal is misunderstood: "Do we weigh the aqueous phase in biodiesel obtaining?" 
(Biodiesel is the non aqueous phase) 

• Hypothesis making: "Will the soap be obtained before if the temperature is higher?" 

• Requesting a word meaning clarification: "Is ethanol the same as 96 9 alcohol?" 

From these initial codes, successive networks may be made. These networks allows us to establish different 
types of relationships among the codes: association, causality, contradiction. Likewise, the encoding process, 
itself, requires splitting and merging in order to create a code list. The quotation analysis is then followed by an 
encoding by list. Code assignation is not a one-to-one relation, as several codes can be related to a single 
quotation. During the entire process a crucial role should be assigned to the memos as tools to guide the 
encoding process. 

At the end, this process creates a network consisting of: 

• Codes, which are the most simple categories in SGQs classification 

• Code families, which bring together all codes showing significant educational relationships 

• Supercodes, which display higher levels of generality and abstraction,and more general issues in in LLEs 
characterization. 

Document 1 (complementary materials) shows the encoding framework and explains the meaning of 
established categories. These codes, super codes and code families can be displayed by means of a network 
which shows the meaningful relationships among them (Document 2 in complementary materials). In order to 
improve the display of this theoretical framework, the network from Atlas.ti is later transformed by means of 
CmapTools®. This software is normally used to make and share conceptual maps, and its graphic resources 
facilitate the expression of hierarchical and quantitative relationships among categories. 

To identify the most interrelated codes, those that share a greater number of common quotations can be 
selected by means of the query tool in Atlas.ti, which allow to recover the co-occurrence among codes. This 
way, for each code, the ones which share at least a quotation are displayed (Document 3 in complementary 
materials). 

By means of a simplified encoding framework, a quantitative approach was applied as an example to compare 
the LLEs of similar groups for two consecutive courses. 


3 RESULTS 

3.1 Qualitative approach: making an encoding framework to describe LLEs 

The encoding process of the SGQs from the 25 analyzed laboratory sessions leads to the proposed network 
shown in figure 1. (Codes, code families and supercodes are described in complementary materials, document 
1; likewise, the network from Atlas.ti is shown in document 2). 
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Fig. 1. Proposed encoding framework from SGQs by means of CmapsTools. 


In this framework (Figure 1), three main realms (matching with supercodes) can be distinguished: 

a) SGQs concerning conceptual and methodological issues. These are classified from three different viewpoints. 
According their purpose, several code families may be identified: 

• Questions intended to receive feed-back. They usually aim to confirm facts or manipulative processes 
before or after doing laboratory operations(code family 1.1.1.) 

• Questions seeking conceptual or methodological information or clarification, (code family 1.1.2.) Often, 
these questions consist of "What to do" or "How to do". 

• Questions which are intended to ask for guidance about "how to do" and "what to do" referred to any 
experimental process (code family 1.1.3.) 

b) Other minority but meaningful questions involve a participative and active attitude regarding laboratory 
organization and development, as well as an interest in delving further into conceptual and methodological 
issues (supercode 2 in figure 1). They are, perhaps, the most representative of autonomy development. 

c) Lastly, some relevant code families are related to infrastructural issues: safety and environmental care; 
organization and availability of material; and proper equipment use and functioning (supercode 3 in figure 1) 

The meaning of the codes included in the first category (supercode 1) can be enriched taking into account the 
available information from the learning context, (code family 1.2. in figure 1). In that sense, these relationships 
may reveal learning deficiencies as the result of poor engagement in the subject-matter, inadequate 
coordination between theoretical and experimental contents, etc. Similarly, some questions reflect a lack of 
understanding of the purpose and goals of general and specific laboratory activities, (code family 1.3. in figure 
1 ). 
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3.2 Descriptive quantitative approach 

To show a quantitative approach to SGQs analysis, several simplified categories combining the defined codes 
has been proposed in table 1. 


Simplified categories 

Categories from encoding framework (fig. 1) 

1. Infrastructural and 
management issues. 

3. Laboratory organization and management. 

3.1. Safety and environmental care. 

Wastes disposal and cleaning methods. 

Asking for safety rules. 

3.2. Use of laboratory instrumentation, glassware and reagents. 

3.2.1. Availability of materials and reagents. Difficulty to locate them. 

3.2.2. Specific amounts of reagents in qualitative tests. 

3.3. Planning and organization of laboratory sessions. 

2. Available information 
from laboratory work 
context and everyday life. 

1.2. According to the context in which information is requested 

1.2.1. Available from activity context: laboratory handbook, prelab 
activities, etc. 

1.2.2. Available from academic context. 

1.2.3. Everyday life. Non specific manipulations of chemistry laboratory. 

3. No available 

information from 

laboratory work context. 

1.2.4. Conceptual or methodological information where previous 
knowledge is unlikely. 

4. "How to do" questions. 

1.1.3.1. "How to do" stages of laboratory activity. 

1.1.3.3. "How to do" specific issues related to laboratory 
manipulations, including apparatus and equipment use. 

1.1.3.4. Other methods: calculations, making graphs, etc." stages of 
laboratory activity. 

5. Feed-back at a later 
stage. 

1.1.1. Receiving feed-back. 

1.1.1.2. Requesting approval after completing specific stage of a 
laboratory activity. 

1.1.1.3. Confirming expectations of final results. 

1.1.1.6. Requesting solutions or explanations for unexpected or 
surprising facts. 

6. Feed-back in advance. 

1.1.1. Receiving feed-back. 

1.1.1.1. Confirmatory questions regarding specific experimental stages 
or facts. 

1.1.1.4. Requesting approval to carry out laboratory task sequence. 

1.1.1.5. Knowing laboratory tasks goals. 

7. Task goals 

misunderstanding. 

1.1.3.2. "What to do" questions. 

1.5. According to evidence of poor understanding regarding activity 
purpose and goals. 

1.5.1. Related to specific issues. 

1.5.2. Related to total understanding of purpose] and goals of 
laboratory tasks. 

8. Reflecting and 

transforming knowledge. 
Participation. 

1.1.2. Reflecting and inquiring about laboratory activity or putting in 
proper context. 

1.1.2.1. Conceptual issues. 

1.2.1.1.1. Concepts definition and clarification. 

1.2.1.1.2. Relation with social and professional context. 

1.1.2.2. Methodological issues. 

1.1.2.2.1. Asking about goals of applied methods. 

1.1.2.2.2. Typical aspects of the scientific methods. 

1.1.2.2.3. Developing applied laboratory methods. 

2. Participation. Favourable attitudes toward learning. Proposal for changes 
and improvements in laboratory session organization and development. 


Table 1. Simplified categories and their correspondence with the ones from encoding framework 
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These simplified categories have been applied to interpret year 2009 and 2010 SGQs analysis (Fig.2) 

i.mafhjcuu amaieBmcrt cmntlfetiw approach t SGQi In3009 and 3010 



Fig. 2. SGQs occurrence (ratios respect the total amount of questions) in year 2009 and 2010 experiences for 

simplified categories 


The quantitative significance of certain identified codes related to infrastructural issues or operational 
knowledge with which the students have no previous experience should be outlined. When the students' 
attention is focused on these concerns, higher cognitive level thinking becomes difficult. This problem is a well- 
known feature of LLEs. According to Byrne [16], during laboratory session, students may be subjected to a large 
amount of heterogeneous and poor, hierarchically- structured information inputs which are considered 
"noise". These inputs are rarely important, and mask the essential meaning ("signal") of the laboratory work. 

In addition, the high incidence of the following types of questions should be highlighted: 

• Those intended to verify that real results match the expected results, or that the operation has been 
performed correctly. 

These types of questions can be interpreted as an indication of low degree of autonomy. A significant number 
of students do not conceive the experimental observations as a fact to be interpreted, but rather as the 
consequence of something which may be well or badly done, according to a "cookbook" approach. 

• Methodological questions regarding the experimental process and questions concerning specific 
laboratory manipulations. These often involve operations from everyday life. 

• Questions requesting information available in the laboratory work context. 

The item 1.2.4.: "Conceptual or methodological information where previous knowledge is unlikely" shows a 
very high score, but must be analyzed in light of the fact that the students were clear beginners with regard to 
laboratory work. In fact, the use of ordinary material and equipments were unknown to the majority of 
students. 

A quantitatively significant and closely linked set of items (Fig.3) can be observed when the code's 
cooccurrence is analyzed (document 3 in complementary materials). Its educational meaning can be 
interpreted a signal of alarm regarding the students' total understanding of laboratory task meaning and its 
function in the subject matter context. 
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Fig. 3. Set of codes with the stronger linkage and guantitative relevance in our educational context. 


If the results from 2009 and 2010 are compared, important similarities in quantitative main codes and links can 
be appreciated. Nevertheless, as an example of the proposed tool usefulness, at least two relevant differences 
may be observed: The rate questions/student is much higher, and the triple linkage (1.1.1.1)-(1.2.1.)- (1.3.1) 
especially strong in academic year 2010. This fact reveals that little attention has been paid to prelab activities 
(Indeed, a significant decrease was observed from year 2009 to 2010 concerning the online tests marks 
associated to each one of prelab activities). 


4 CONCLUSIONS 

The theoretical framework established from the SGQ analysis provides noteworthy categories in line with 
categories indicated in the literature (Hofstein, Nahum & Shore, 2001). Nevertheless, the proposed theoretical 
framework includes categories and relationships among them which can be a valuable contribution in order to 
a more detailed and deepen LLEs knowledge. In that respect, several critical issues can be distinguished: 

• Some code families are clear noise sources. More attention should be paid to their origin, and measured 
taken to guarantee minimum laboratory skills before taking a university course in general chemistry. It is 
paradoxical that great effort has been devoted to research on misconceptions, while alternative 
frameworks, etc., and hands-on competences have been undervalued, even with regard to everyday and 
simple manipulative skills. 

• More effective display of information concerning waste disposal, material and reagents location etc. 
could help improve laboratory activities. Indeed, improvements in laboratory management must be linked 
to careful design of all relevant educational resources in order to achieve absolute consistency in 
information received by students regarding organizational and infrastructural aspects such as schedules, 
environmental protection, safety rules etc. ((e.g., many SGQs reflect inconsistency in labelling of reagents or 
glass materials and information provided in laboratory handbook, thus causing unnecessary confusion and 
loss of time. 


• The great frequency of SGQs aimed at receiving feed-back-- usually as "confirmatory" questions-- 
together with the poor cognitive level of students regarding conceptual or methodological inquiry is not 
surprising. This evidence should encourage us to promote the methodological and organizational changes 
that incorporation into the European Space of Higher Education demands. 


In summary, from the standpoint of educational practice, this methodology may be particularly useful for 
implementing experimental syllabi or methodological innovations in a learning context. These results lead us 
to argue for a radical transformation in the role of laboratory work in learning. Indeed, lab work must be 
integrated into a coherent process of constructing knowledge, beyond its traditional role of testing theory and 
hands-on skills development. Furthermore, a number of problems related to management laboratory issues 
can be detected by means of the impact of items in category 3 in encoding framework (Fig. 1). This information 
may be useful feed-back for laboratory instructors. 


In the research field, the proposed methodology may be useful as a qualitative way to be used together with 
quantitative tools in order to enhance the significance and internal validity of these tools. SGQ analysis can be 
an effective means of attaining a realistic and accurate characterization of LLEs in experimental control groups 
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in educational research design, beyond their simple and aprioristic distinction based on the instructional 
proposals. 


5 COMPLEMENTARY MATERIALS 

The following complementary materials can be retrieved from: 

https://poliformat.upv.es/access/content/user/19877419/ELSIN 2010/ 

• Document 1: Defined codes, supercodes and code families from five years long experience. 

• Document 2: Network from Atlas.ti 

• Document 3: Codes' cooccurrence 
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